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PREDICTIVE DISPLAY FOR SYSTEM HAVING DELAYED FEEDBACK OF A 

COMMAND ISSUED 



PREDICTIVE DISPLAY FOR A SYSTEM HAVING DELAYED 
FEEDBACK OF A COMMAND ISSUED 

FIELD AND BACKGROUND OF THE INVENTION 

The present invention relates to a predictive display and, in particular, it 
5 concerns a predictive display for a system having delayed feedback of a 
command issued. 

By way of introduction, remotely operated devices generally require 
feedback from the device to the operator, so that the operator can decide on 
what action to take next. This is generally achieved by having a video camera 

10 or other similar apparatus disposed on the device. The video camera then relays 
feedback of the position of the device with respect to its surroundings. 
However, there is a time delay, typically of up to several seconds, between the 
time when the operator gives a movement command to the device and when the 
operator receives feedback with respect to the completion of the movement 

15 command. The delay is typically due to several factors. First, there may be a 
transmission time delay due to communication from the operator to the device 
and from the device to the operator. Second, there is a time delay due to the 
time taken by the device to carry out the movement command. Therefore, the 
operator needs a prediction of the final position of the device at the time the 

20 movement command is being created to be able to decide whether the 
movement command is appropriate. A solution taught by the prior art is to 
superimpose an icon such as a motion vector or arrow to represent the 



movement command. The beginning of the vector or arrow denotes the current 
position and the end of the vector or the arrowhead denotes the predicted 
position after the movement command has been carried out or the predicted 
position after a predetermined time interval. A shortcoming of the 
5 aforementioned system is due to the system not being intuitive to the operator. 
An operator needs to pay close attention to the arrow or vector. A further 
shortcoming of the aforementioned system is that the image of the view from 
the device as well as the superimposed vector are updated as the device moves, 
causing further confusion to the operator as well reduced operator performance. 
10 There is therefore a need for an intuitive predictive display for a system 

having delayed feedback of a command issued. 

SUMMARY OF TH E INVENTION 

The present invention is a predictive display for a system having delayed 
feedback of a command issued. 
15 According to the teachings of the present invention there is provided a 

method to provide feedback to an operator of a device having feedback delay, 
comprising the steps of: (a) displaying a first image of a view from the device, 
the device being at a first position; (b) issuing a movement command to cause a 
desired movement of the device to a second position; and (c) displaying a 
20 second image of a predicted view from the device at the second position prior 
to the operator receiving real feedback of the movement command. 
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According to a further feature of the present invention, the second image 
is based upon at least part of the first image. 

According to a further feature of the present invention, the second image 
includes a filler section outside of the at least part of the first image. 
5 According to a further feature of the present invention, the filler section 

includes a pattern. 

According to a further feature of the present invention, the filler section 

includes a repetitive pattern. 

According to a further feature of the present invention, the filler section 
1 0 includes historic image data of the predicted view. 

According to a further feature of the present invention, there is also - 
provided the step of displaying a third image of an actual view from the device 
at the second position. 

According to a further feature of the present invention, there is also 
1 5 provided the step of limiting the movement command to ensure that the second 
image can be based upon at least part of the first image. 

According to a further feature of the present invention, the step of 
issuing the movement command and the step of displaying the second image, 
occur substantially at the same time. 
20 According to a further feature of the present invention, the step of 

displaying the first image is performed by displaying the first image on a 
screen, the screen having a frame disposed thereon, the first image being 
disposed substantially within the frame and wherein the step of displaying the 
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second image is performed by displaying the second image on the screen such 
that, the second image includes substantially all image elements of the first 
image. 

According to the teachings of the present invention there is also 
5 provided, a feedback system for an operator of a device having a camera, 
comprising: (a) a control arrangement configured for issuing a movement 
command to cause a desired movement of the device from a first position to a 
second position; and (b) a display configured for: (i) displaying a first image of 
a view from the device, the device being at a first position; and (ii) displaying a 
10 second image prior to the operator receiving real feedback of the movement 
command, the second image being a predicted view from the device at the 
second position, the second image being based upon at least part of the first 
image. 

According to a further feature of the present invention, the display is 
15 further configured for displaying a third image of an actual view from the 
device at the second position. 

BRIEF DESCRIPTION OF THE DRAWINGS 

The invention is herein described, by way of example only, with 
reference to the accompanying drawings, wherein: 
20 Fig. 1 is a schematic view of a remotely operated system having a 

display that is constructed and operable in accordance with a preferred 
embodiment of the present invention; 



Fig. 2 is a view of the display of Fig. 1 showing an image of a predicted 
view from the device due to a movement command; 

Fig. 3 is a view of the display of Fig. 1 showing an image of a predicted 
view from the device due to a movement command, the image including a 
repetitive pattern "filler" section that is constructed and operable in accordance 
with a first alternate embodiment of the present invention; 

Fig. 4 is a view of the display of Fig. 1 showing an image of the actual 
view from the device after the movement command has been completed and 
the image of the view from a camera at the new position has been received; 

Fig. 5 is a view of the display of Fig. 1 showing the image of a predicted 
view from the device due to a forward or zoom-in movement command; 

Fig. 6 is a view of a display prior to issuing a movement command that 
is constructed and operable in accordance with a second alternate embodiment 
of the present invention; 

Fig. 7 is a view of the display of Fig. 6 showing an image of a predicted 
view from the device due to a movement command; and 

Fig. 8 is a view of the display of Fig. 6 showing an image of the actual 
view from the device after the movement command has been completed and 
the image of the view from a camera at the new position has been received. 



DESCRIPTION OF THE PREFERRED EMBODIMENTS 

The present invention is a predictive display for a predictive display for 
a system having delayed feedback of a command issued.. 

The principles and operation of a predictive display for a system having 
5 delayed feedback of a command issued according to the present invention may 
be better understood with reference to the drawings and the accompanying 
description. 

Reference is now made to Fig. 1, which is a view of a remotely operated 
system 10 having a display 12 that is constructed and operable in accordance 

10 with a preferred embodiment of the present invention. Remotely operated 
system 10 also has a remotely controlled device 14 and an operator module 22. 
Device 14 has disposed thereon, a camera 18 and' a transceiver 20. Device 14 is 
typically a remotely operated vehicle or surveillance camera arrangement. It 
will be apparent to those skilled in the art that when device 14 is a surveillance 

15 camera, movement of device 14 is represented by movements and zooming of 
camera 18. However, it will be apparent to those skilled in the art that 
device 14 could be any device requiring operator control via a predictive 
display. Additionally, it will be appreciated by those ordinarily skilled in the art 
that remotely operated system 10 includes systems whereby the operator is 

20 disposed within or on top of device 14. Operator module 22 includes 
display 12, a movement controller 16, a processor 24 and a transceiver 26. 
Camera 18 is configured to send images of views from device 14 via 
transceiver 20 to transceiver 26. Transceiver 20 and transceiver 26 typically 
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define a wireless communication link. However, it will be apparent to those 
skilled in the art that processor 24 and device 14 may be connected using a 
communication cable or other communication link. Images received by 
transceiver 26 are processed by processor 24 and then displayed by display 12. 
5 The frequency of images received by transceiver 26 is typically frequent 
enough, such that, the view from camera 18 is viewed by the operator as a 
"continuous" video or a series of discrete images. The operator issues 
movement commands to device 14 using movement controller 16. Movement 
controller 16 is typically includes a joystick or pointing device such as a mouse 
10 or a steering wheel as well as an other movement control devices capable of 
instructing regarding direction and steering as well as zooming a lens of 
camera 18. Movement commands are sent via processor 24, transceiver 26 and 
transceiver 20 to an actuator 27 which is configured to move device 14. As 
camera. 18 is generally disposed in a known relationship with respect to 
15 device 14, images of views captured by camera 18 define the position and 
orientation of device 14 in relation to the surroundings. Display 12 shows an 
image 28 of a view from device 14 while device 14 is at a first position. It 
should be noted that image 28 is updated constantly to reflect views captured 
by camera 18 of the first position, taking into account the delay between 
20 camera 18 capturing the view and predictive display 12 displaying image 28. 

By way of introduction, due to the time delay between the operator 
issuing a movement command to cause a desired movement of device 14 from 
the first position to a second position and the operator receiving feedback of 
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that movement command, the operator needs a way to visualize the final 
position due to the movement command at the time the movement command is 
issued. The desired movement may be linear, rotational or a more complex 
movement. Therefore, according to the present invention, a predicted image of 
5 the view from device 14 at the second position is displayed on display 12 
typically at the time the operator issues the movement command, which is prior 
to the operator receiving real feedback of the actuated movement command. 
Real feedback is defined herein as an actual image of a view from camera 18 at 
the second position and not a predicted image of a view from camera 18 at the 
1 0 second position. 

Reference is now made to Fig. 2, which is a view of display 12 showing 
an image 30 of a predicted view from device 14 due to a movement command. 
Remotely operated system 10 is configured such that image 30 of the predicted 
view from device 14 at the second position is displayed on display 12 at 
15 substantially the same time that a movement command is issued. Therefore, 
image 30 is displayed prior to the operator receiving real feedback of the 
actuated movement command. Image 30 is based upon at least part of image 28 
(Fig. 1). It will be appreciated by those skilled in the art that as movement 
controller 16 is actuated, the. image displayed on predictive display 12 is altered 
20 to reflect the current predicted view from camera 18 after the movement 
command is complete. The current predicted view from camera 18 after the 
movement command is complete, is formed by manipulating image 28, for 
example by moving or scaling or rotating image 28, as appropriate, such that, 



the center of an image of a view from camera 18 at the second position 
corresponds to the center of predictive display 12. Scaling of image 28 is 
necessary where forward and reverse movement commands are issued. Other 
manipulation of image 28 is performed to reflect the desired results. Scaling of 
5 image 28 is illustrated with reference to Fig. 5. Therefore, as movement 
controller 16 .is actuated, the image displayed by predictive display 12 
gradually changes from image 28 to image 30. Therefore, there is a plurality of 
images which are displayed on predictive display 12, which represent the 
actuating of movement controller 16 in order to move device 14 from the first 
10 position to the second position. In the example of Fig. 2, the movement 
command given by the operator to move device 14 from the first position to the 
second position was to turn device 14 in a left direction. Therefore, in this 
example, the right portion of image 30 is formed by using most of image 28 
except for part of the right portion of image 28. The left portion of image 30 
15 has a blank "filler" section 32 indicative of image data which cannot be derived 
from image 28. It should be noted that as the operator moves the steering 
control of movement controller 16 to the left, image 30 will change to include 
gradually less of the right portion of image 28 and more of "filler" section 32. 
It will be appreciated by those skilled in the art that other operations including 
20 lateral, rotational, scaling and other image manipulations may be performed by 
processor 24 on image 28 to create image 30 depending on the desired 
movement command. Additionally, image 30 may be created by processor 24 
using advanced image processing techniques to produce an image which is 
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based upon image 28 as well as other images of prior views stored by remotely 
operated system 10. It should be noted that image 30 is created by altering 
image 28 to reflect the predicted view from camera 18 when device 14 is at the 
second position and not by superimposing an arrow or vector on image 28 to 
5 show where the second position is. Typically, processor 24 instructs 
actuator 27 to automatically execute the movement commands as steering 
occurs. Optionally, the movement command is manually confirmed to 
actuator 27 of device 14 to commence actual execution of the movement 
command. Actuator 27 then completes the desired movement of device 14 to 
10 the second position. Although image 30 continually reflects an image of the 
predicted view at the second position, image 30 is continually updated by 
views captured by camera 18 even while camera 18 is moving. 

As an optional feature, the movement command is limited to ensure that 
image 30 can be based upon at least part of image 28 so that the operator will 
15 be able to visualize at least partially the effect of the movement command. If 
the movement command is such that the second position cannot be described 
using image 28 then image 30 will be blank unless other historic data is 
available. 

Reference is now made to Fig. 3, which is a view of display 12 showing 
20 an image 34 of a predicted view from device 14 due to a movement command. 
Image 34 includes a "filler" section 36 that is constructed and operable in 
• accordance with a first alternate embodiment of the present invention. Filler 
section 36 is typically a static pattern which is manipulated in the same way as 
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image 34, for example, by translation, rotation or scaling. Filler section 36 is 
generally a repetitive pattern such as a grid. Optionally, additional historic 
image data of the predicted view is used to fill in the filler section of image 34 
instead of using a blank filler or a repetitive pattern filler. The historic image 
5 data is based on previous images at or close to the second position. 

Reference is now made to Fig! 4, which is a view of display 12 showing 
an image 38 of the actual view from device 14 after the movement command 
has been completed and the image of the view from camera 18 at the second 
position have been received by operator module 22. It should be noted that 
10 image 38 is formed by gradual changes to image 30 as camera 18 captures 
views as camera 18 moves. 

Reference is now made to Fig. 5, which is a view of display 12 showing 
an image 40 of a predicted view from device 14 due to a forward or zoom-in 
movement command. It is seen that image 40 is an enlarged version of 
15 image 28 (Fig. 1). Therefore, image 40 reflects the forward or zoom-in 
movement command, which brings the objects of image 28 closer. 

Reference is now made to Fig. 6 and Fig. 7. Fig. 6 is a view of a 
display 42 prior to issuing a movement command that is constructed and 
operable in accordance with a second alternate embodiment of the present 
20 invention. Fig. 7 is a view of display 42 showing an image 44 of a predicted 
view from device 14 due to a movement command. Display 42 has a centrally 
disposed frame 46. Frame 46 is either an image or a permanent fixture which is 
disposed on display 42. Before a movement command is issued, an image 48 of 
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an actual view from device 14 is displayed wholly within frame 46. At this 
point, device 14 is at a first position. The operator then issues a movement 
command, for example, to move device 14 to the left to a second position. As 
the movement command is issued, image 44 is displayed on display 42. 
5 Image 44 is image 48 which is moved to the right on display 42 and partially 
outside of frame 46 to reflect the steering to the left by the operator. However, 
substantially all image elements of image 48 are included within image 44. The 
term "substantially all image elements are included" is defined herein to 
include, for example, when some of the details of image 48 are lost due to 
10 scaling of image 48 to produce image 44. 

Reference is now made to Fig. 8, which is a view of predictive 
display 42 showing an image 50 of the actual view from device 14 after the 
movement command has been completed and the image of the view from 
camera 18 at the second position has been received. 
15 It will be appreciated by persons skilled in the art that the present 

invention is not limited to what has been particularly shown and described 
hereinabove. Rather, the scope of the present invention includes both 
combinations and sub-combinations of the various features described 
hereinabove, as well as variations and modifications thereof that are not in the 
20 prior art which would occur to persons skilled in the art upon reading the 
foregoing description. 
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WHAT IS CLAIMED IS: 

1. A method to provide feedback to an operator of a device having 
feedback delay, comprising the steps of: 

(a) displaying a first image of a view from the device, the device being at a 
first position; 

(b) issuing a movement command to cause a desired movement of the 
device to a second position; and 

(c) displaying a second image of a predicted view from the device at said 
second position prior to the operator receiving real feedback of said 
movement command. 

2. The method of claim 1 , wherein said second image is based upon at least 
part of said first image. 

3. The method of claim 2, wherein said second image includes a filler 
section outside of said at least part of said first image. 

4. The method of claim 3, wherein said filler section includes a pattern. 

5. The method of claim 4, wherein said filler section includes a repetitive 



pattern. 
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6. The method of claim 3, wherein said filler section includes historic 
image data of said predicted view. 

7. The method of claim 1, further comprising the step of: 

(d) displaying a third image of an actual view from the device at said second 
position. 

8. The method of claim 1, further comprising the step of: 

(d) limiting said movement command to ensure that said second image can 
be based upon at least part of said first image. 

9. The method of claim 1, wherein said step of issuing said movement 
command and said step of displaying said second image, occur substantially at 
the same time. 

10. The method of claim 1, wherein said step of displaying said first image 
is performed by displaying said first image on a screen, said screen having a 

. frame disposed thereon, said first image being disposed substantially within 
said frame and wherein said step of displaying said second image is performed 
by displaying said second image on said screen such that, said second image 
includes substantially all image elements of said first image. 

11. A feedback system for an operator of a device having a camera, 
comprising: 
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(a) a control arrangement configured for issuing a movement command to 
cause a desired movement of the device from a first position to a second 
position; and 

(b) a display configured for: 

(i) displaying a first image of a view from the device, the device 
being at a first position; and 

(ii) displaying a second image prior to the operator receiving real 
feedback of said movement command, said second image being a 
predicted view from the device at said second position, said 
second image being based upon at least part of said first image. 

12. The system of claim 11, wherein said display is further configured for 
displaying a third image of an actual view from the device at said second 
position. 

Mark M. Friedman 
^Advocate, Patent Attorney 
Beit Samueloff 
7 Haomanim 
67897 Tel Aviv 
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